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Before  an  image  is  stored  or  transmitted,  we  have  access  to  the  original  and  the  distorted 
versions.  The  enhanced  codec  is  compared  to  the  original  block  by  block  to  determine  which  blocks 
have  been  improved  by  the  enhancement  These  blocks  are  then  flagged  for  post-processing  in  a 
way  that  is  compliant  with  the  JPEG  standard  and  adds  nothing  to  the  compressed  image's 
bandwidth.  The  end  result  is  a  compressed  image  that  can  be  decompressed  on  any  standard  JPEG 
decompressor,  but  that  can  be  enhanced  by  a  sophisticated  decompressor. 

For  the  comparison  of  the  original  and  enhanced  images,  we  have  been  developing  a  new 
vision  model  that  is  specifically  tailored  to  the  detection  of  errors  that  occur  within  or  between  two 
JPEG  codec  blocks.  Previous  filter  models  have  been  restricted  from  using  a  large  number  of  filters 
due  to  computational  constraints  which  we  avoid  by  focusing  the  model  on  a  tiny  spatial  area  of 
8x16  pixels.  Further,  features  of  human  vision  that  have  been  included  in  previous  models  (color, 
temporal,  stereo  etc.)  are  not  needed  for  this  more  focused  problem.  Issues  that  have  not  been 
completely  addressed  by  previous  models,  such  as  masking  effects,  are  tractable  and  the  model  is 
more  applicable  to  JPEG  compression. 
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A  tiny  vision  model 

A  number  of  researchers  have  been  developing  systems  to  automatically  assess 
the  visual  fidelity  of  distorted  images  with  varying  degrees  of  success.  The  goal  of  this 
research  is  to  develop  an  improved  fidelity  metric  that  is  specifically  tailored  for  the 
detection  of  errors  that  occur  in  a  single  JPEG  codec  block,  or  between  two  JPEG  codec 
blocks.  Previous  filter  models  have  been  restricted  from  using  a  large  number  of  niters 
due  to  computational  constraints.  By  focusing  the  model  on  a  tiny  spatial  area  of  8x8  or 
8x16  pixels,  it  will  be  possible  to  have  a  huge  variety  of  filters  without  an  excessive 
computational  burden.  Further,  many  of  the  features  of  human  vision  that  have  been 
included  in  previous  models  (color,  temporal,  stereo  etc.)  will  not  be  needed  for  this  more 
focused  problem.  By  limiting  the  problem,  issues  that  have  not  been  completely 
addressed  by  previous  models,  such  as  masking  effects,  will  become  more  tractable.  Hie 
model  will  also  have  more  direct  applicability  to  improving  JPEG  compression,  as  will  be 
discussed. 

Precomputing  and  encoding  enhancement  instructions  for  inclusion  with 
compressed  image  files 

There  has  been  much  recent  work  in  the  field  of  image  enhancement  for  digital 
images.  Traditional  techniques  such  as  filtering  and  edge  enhancement  have  been  applied 
to  restoring  images  that  have  been  distorted  due  to  lossy  image  compression.  However, 
these  techniques  have  ignored  a  unique  feature  that  can  be  exploited  when  working  with 
digital  compression.  Before  the  image  is  stored  or  transmitted,  the  sender  has  access  to 
both  the  original  and  the  distorted  images.  Enhancement  can  be  selectively  applied  and 
assessed  before  the  image  is  stored  or  transmitted,  and  successful  enhancement 
instructions  can  be  included  with  the  image  file. 

In  this  paper  we  present  a  technique  that  can  be  used  to  efficiently  include  a 
spatial  map  of  enhancement  instructions  with  compressed  image  files.  The  method  we 
describe  can  be  used  to  produce  files  that  are  compliant  with  the  JPEG  standard,  so  that 
the  file  can  be  decompressed  with  a  standard  decoder.  An  advanced  decoder  that  can  read 
the  enhancement  instructions  can  take  the  same  file  and  produce  an  enhanced  image. 

Before  an  image  is  stored  or  transmitted,  we  have  access  to  the  original  and  the 
distorted  versions.  The  enhanced  codec  is  compared  to  the  original  block  by  block  to 
determine  which  blocks  have  been  improved  by  the  enhancement  These  blocks  are  then 
flagged  for  post-processing  in  a  way  that  is  compliant  with  the  JPEG  standard  and  adds 
nothing  to  the  compressed  images’  bandwidth.  A  single  JPEG  coefficient  is  adjusted  so 
that  the  sum  of  the  coefficients  contains  the  flag  for  post-processing  as  the  parity  of  the 
block.  Half  of  die  blocks  already  have  the  correct  parity.  In  the  other  blocks,  a  coefficient 
that  is  close  to  being  half  way  between  two  values  will  be  chosen  and  rounded  in  die 
other  direction.  This  distorts  the  image  by  a  very  tiny  amount.  The  end  result  is  a 
compressed  image  that  can  be  decompressed  on  any  standard  JPEG  decompressor,  but 
that  can  be  enhanced  by  a  sophisticated  decompressor. 

In  addition  to  adding  single  bits  of  data  in  the  parity  of  a  block,  we  can  use  a  more 
complicated  parity  scheme  to  add  more  information.  For  example,  the  parity  of  a  block 
can  be  adjusted  so  that  the  sum  of  the  coefficients  modulo  3  is  in  one  of  three  possible 
states.  The  three  states  can  represent  three  alternate  processing  schemes,  instructions  for 
which  of  the  blocks  edges  to  smooth,  or  other  multi-bit  information.  We  present 
equations  for  estimating  the  error  introduced  into  the  image  by  the  parity  forcing  method, 
and  we  compare  our  estimates  to  computer  simulations  and  actual  image  data. 
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What  are  the  spatial  and  temporal  limits  of  relative  motion  discrimination? 
Motion  of  very  small  amplitudes  (5  sec  or  less)  can  be  detected  when  the  frequency  of 
motion  is  optimal  (2-4  Hz).  Previous  research  has  shown  that  under  some  conditions,  the 
relative  motion  of  two  feature  can  be  judged  whenever  the  amplitude  of  motion  is 
sufficient  for  detection.  In  these  experiments,  we  measure  the  spatial  and  temporal 
properties  of  relative  motion  discrimination. 

We  used  two  sinusoidally  oscillating  features  (dots  or  lines)  to  determine  the 
temporal  frequency  response  function  of  relative  motion  discrimination.  We  measured 
both  the  threshold  amplitude  required  for  detecting  and  discriminating  motion.  We  also 
measured  the  transducer  functions  that  relate  amplitude  to  detectability  and 
discriminibility.  We  determined  the  slope  of  the  transducer  functions,  and  found  a 
saturating  non  linearity  for  high  temporal  frequency  motion. 

The  subject  was  presented  with  two  dots  that  either  oscillated  inphase  (the  dots 
moved  in  the  same  direction)  or  in  counterphase  (the  dots  moved  in  opposite  directions). 
We  varied  the  temporal  frequency  of  oscillation,  the  separation  between  the  two  features, 
the  lighting  conditions,  the  amptitudes  of  oscillation  and  the  distance  from  the  fovea  of 
the  two  features.  When  the  temporal  frequency  of  the  presented  stimuli  is  optimal  (~4 
Hz),  the  visual  system  is  remarkably  adept  at  discriminating  relative  motion,  if  the  motion 
can  be  detected,  its  type  can  be  discriminated.  Contrary  to  previous  findings,  a  temporal 
frequency  of  more  than  8  Hz  degrades  thresholds  relative  to  detection  and  at  14  Hz 
increases  them  more  than  5-fold. 


